Predicting the Performance of Parsing with Referential Translation Machines

Referential translation machine (RTM) is a prediction engine used for predicting the performance of natural language processing tasks including parsing, machine translation, and semantic similarity pioneering language, task, and domain independence. RTM results for predicting the performance of parsing (PPP) in out-of-domain or in-domain settings with different training sets and types of features present results independent of language or parser. RTM PPP models can be used without parsing using only text input and without any parser or language dependent information. Our results detail prediction performance, top selected features, and lower bound on the prediction error of PPP. 1. Predicting Parsing Performance with Referential Translation Machines Training parsers and parsing can be computationally costly and labeled data scarce or expensive to obtain. Predicting the performance of parsing (PPP) can be useful for parsing technology, for filtering sentences in noisy domains such as informal text or speech, for estimating the effort for understanding text, for determining whether a sentence is well-formed and meaningful enough to send to other natural language processing (NLP) tasks such as machine translation in an NLP pipeline. PPP involves finding a function f: f(MP,Dtrain, S [, SP ′]) ≈ eval(SP ′, SP) (1) where •MP is a parsing model built using Dtrain for training, • Dtrain is the set of training sentences and Dtest is test data, © 2016 PBML. Distributed under CC BY-NC-ND. Corresponding author: [email protected] Cite as: Ergun Biçici. Predicting the Performance of Parsing with Referential Translation Machines. The Prague Bulletin of Mathematical Linguistics No. 106, 2016, pp. 31–44. doi: 10.1515/pralin-2016-0010. PBML 106 OCTOBER 2016 • SP ′ refers to parsing output obtained on S ∈ Dtest and its reference is SP, • eval returns the bracketing F1 score by EVALB (Sekine and Collins, 1997) implementing the PARSEVAL F1 measure, • the performance of MP, which use Dtrain, is being predicted for input S, • f predicts the value of the eval function to approximate the performance without the reference SP given a training set and a test set not necessarily after training a parsing model or parsing. Ravi et al. (2008) predict the performance ofCharniak and Johnson (CJ) parser (Charniak and Johnson, 2005) using text-based and parser-based features, and additional parser output (Bikel parser (Bikel, 2002)). Additional parser output is used as a reference to obtain a feature with bracketing F1 score. In Section 3.3, we achieve better results using only textual features and obtain similar results without any parser or label dependent information or without an additional parser or its output. Each referential translation machine (RTM) (Biçici and Way, 2015) model is a data translation prediction model between the instances in the training set and the test set, and translation acts are indicators of the data transformation and translation. RTM effectively judges monolingual and bilingual similarity while identifying translation acts between any two data sets with respect to a reference corpus. RTM allows development of prediction models specially prepared for a given training and test set pair. RTM PPP models are built for each task emerging from training set, test set, and label set obtained from a parser. RTMs achieve top results in machine translation performance prediction (MTPP) in quality estimation task (Biçici et al., 2015b; Biçici, 2016), can achieve better results than open-source MTPP tool QuEst (Shah et al., 2013; Biçici and Specia, 2015), and can achieve top results in semantic similarity prediction tasks (Biçici andWay, 2015). We provide a current picture on PPP detailing prediction performance, top selected features, and lower bound on prediction error of PPP. RTMs judge the quality or the semantic similarity of texts by using relevant retrieved data close to the task instances as interpretants, selected preferably from the same domain. RTM PPP use parallel and monolingual sentences as interpretants, which provide context and data for MTPP system (MTPPS) (Biçici and Way, 2015) to derive features measuring the closeness of the test sentences to the training data, the difficulty of translating them, and the presence of the acts of translation for building prediction models. RTMs present an accurate and language-independent model for NLP performance prediction and provide a parser-independent model, which enables the prediction of the performance of any parser in any language. Figure 1 depicts the workflow for a general RTMmodel and explains themodel building process. Given a training set train, a test set test, and some corpus C, preferably in the same domain, the RTM steps are: 1. select(train, test, C) → I 4. learn(M,Ftrain) → M 2. MTPP(I, train) → Ftrain 5. predict(M,Ftest) → ŷ 3. MTPP(I, test) → Ftest 32 Ergun Biçici Predicting the Performance of Parsing with RTM (31–44) Figure 1. RTM workflow: ParFDA selects interpretants close to the training and test data using parallel corpus in bilingual settings and monolingual corpus in the target language or just the monolingual target corpus in monolingual settings; an MTPPS use interpretants and training data to generate training features and another use interpretants and test data to generate test features in the same feature space; learning and prediction takes place taking these features as input. RTM PPP models use MTPPS to generate features and parallel feature decay algorithms (ParFDA) (Biçici et al., 2015a) for instance selection. The modularity of RTM enables additional knowledge sources to be retrieved by ParFDA, which can be used for deriving additional features to be included before learning and prediction. 2. Statistical Lower Bound on Prediction Error We evaluate the prediction performance with correlation (r), root mean squared error (RMSE), mean absolute error (MAE), and relative absolute error (RAE). Given that ŷ, y ∈ R are the prediction of F1 and the target F1 respectively: